HiFun - A High Level Functional Query Language for Big Data Analytics

We present a high level query language, called HiFun, for defining analytic queries over big data sets, independently of how these queries are evaluated. An analytic query in HiFun is defined to be a wellformed expression of a functional algebra that we define in the paper. The operations of this algebra combine functions to create HiFun queries in much the same way as the operations of the relational algebra combine relations to create relational algebra queries. The contributions of this paper are as follows: (a) defining a formal framework (i.e. HiFun) in which to study analytic queries in the abstract in much the same way as the relational algebra provides the formal framework to study relational algebra queries; (b) showing that each HiFun query can be encoded as a map-reduce job, and also as a SQL group-by query when the data set is a relational database; and (c) defining a formal method for rewriting HiFun queries and, as a case study, showing how our method can be applied in the rewriting of map-reduce jobs and of SQL group-by queries. We emphasize that, although theoretical in nature, our work uses only basic and well known mathematical concepts, namely functions and their basic operations. 1